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ABSTRACT 



This paper critiques current models for evaluation of mental 
health services provided to children and adolescents with emotional 
disturbances and offers an alternative model. The most popular model, 
Donabedian's tripartite model, is described. This model separately evaluates 
three domains: (1) structure (characteristics of the care providers, their 

tools and resources, and the physical/organizational setting) ; (2) process 

(both interpersonal and technical aspects of the treatment process) ; and (3) 
outcome (change in the patient's symptoms and functioning). The paper then 
notes three approaches used to assess these domains: the use of standards and 
guidelines, peer review, and outcomes monitoring. Problems with these 
approaches are identified including: lack of validity data on the importance 
of structure and process indicators, the poor reliability of peer reviews, 
and difficulty in relating outcomes directly to services. The paper proposes 
a model in which quality is viewed strictly as the relationship between 
quality indicators (i.e., structure and process variables) and quality 
validators (i.e., outcomes). This approach is in contrast to viewing quality 
as existing separately within each component. Benefits of this approach 
include better services, use of quality indicators with known validity, and 
better program planning. (Contains 19 references.) (DB) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 
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Introduction 

Consumers, providers, and policymakers all have called for 
the monitoring of quality in order to ensure that lower costs 
do not mean low-quality care. The assessment of quality is 
also a significant element of the continuous quality 
improvement movement, a movement that is crucial for 
improving mental health services. 

Three circumstances inhibit meaningful monitoring of 
quality of care in children's mental health. First, the 
indicators compiled to date do not extensively cover all 
quality domains. Second, research on quality of care in 
children's health services is conspicuously lacking 
(McGlynn, Halfon, & Leibowitz, 1995). Third, the concept 
of quality of care is underdeveloped to the extent that 
current approaches to monitoring quality may fail to assess 
the "goodness" of delivered services. This summary focuses 
on the last concern. 
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The Holy Trinity of Structure, Process, and 
Outcome 

Most approaches to evaluating quality are based to some 
extent on Donabedian's tripartite model of quality. This 
model addresses three domains; structure, process, and 
outcome (Donabedian, 1980). Structure refers to "the 
relatively stable characteristics of the providers of care, of 
the tools and resources they have at their disposal, and of 
the physical and organizational settings in which they work" 
(Donabedian, 1980, p. 81). Five structural categories are 
thought to be important in assessing the quality of service 
structure: (a) Access; (b) Institutional Characteristics;- (c) 
Provider Characteristics; (d) Community Characteristics; 
and (e) Client Characteristics. 

Process focuses primarily on treatment process, including 
interpersonal process factors and technical skill in the 
delivery of services. Interpersonal process refers to the 
therapeutic relationship and rapport, communication, 
information dissemination, and shared decision-making that 
occur as part of treatment. Technical skill encompasses 
knowledge of state-of-the-art intervention techniques, the 
ability to assess which intervention provides the best match 
for the client's problems or diagnosis, and the skill to 
effectively deliver the best matching intervention. 

While Donabedian and other quality-of-care theorists focus 
on technical and interpersonal factors in understanding 
process, additional quality process categories should also be 
considered. Stages-of-treatment-process indicators concern 
the quality of intake and assessment, treatment planning, 
service utilization (e.g., # of sessions/service units, length of 
treatment, and service patterns), and appropriate 
termination. Service process can also refer to barriers to 
treatment, timeliness of treatment, accountability to 
consumers, consumer advocacy, and protection of client 
rights. 

The last component of quality is outcome. Donabedian 
(1980) defined outcome as "a change in the patient's current 
and future health status (symptoms and functioning) that can 
be attributed to antecedent health care" (p. 82). Donabedian 
also included patient attitudes about treatment, such as 
patient satisfaction, health-related knowledge, and 
behavioral change in areas that contribute to health 
problems. Protection of child and family rights and safety 
can also be added as another important outcome category. 
This would include the maintenance of confidentiality, the 
utilization of least restrictive services, and the right to 
receive services known to be effective. 

Approach to MonitoringQuality of Care 

One basic concept appears to underlie all approaches to 
monitoring quality of care: that quality can be examined by 
assessing any one of its three components. This concept is 
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assessing any one of its three components. This concept is 
consistent with Donabedian (1980), who considered the 
evaluation of structure, process, or outcome as all providing 
a similar picture of quality of care. Three approaches have 
been taken to assess quality of care: the use of standards and 
guidelines, peer review, and outcomes monitoring. 

The evaluation of structure and process indicators has been 
based on standards, practice guidelines, and report cards 
developed through literature reviews, general consensus, 
and expert panels. A growing number of organizations and 
agencies have developed or are in the process of developing 
standards, guidelines, and report cards: JCAHO, NCQA, 
AHCPR, CHAMPUS, SAIC, CMHS, AMBHA, and NAMI. 

Peer review is a second approach used to assess quality of 
care process, and more specifically, the assessment of 
appropriateness of care. In a review of the literature, Lee 
Sechrest noted that peer review of mental health care 
process is common (Sechrest, 1987). For example, 
standardized peer review procedures have been used to 
assess appropriateness of psychiatric hospitalization 
(Strumwasser et al., 1991) and quality of outpatient services 
(e.g., Daniels, Kramer, & Mahesh, 1995; Hargrave & Hiatt, 
1995). Moreover, researchers affiliated with the RAND 
Corporation have used a "tracer methodology" involving the 
clinical review of medical records to assess quality of care 
(Wells etal., 1993). 

Finally, outcomes monitoring has grown in popularity over 
the past few years as a way to assess quality (Guadagnoli & 
McNeil, 1994). This method assumes that quality care has 
been delivered if, for example, clients are found to have 
decreased symptoms and/or increased functioning after 
treatment, or clients report they are satisfied with treatment. 

Problems with Current Approaches to Monitoring 
Quality of Care 

Three glaring problems are evident with these approaches to 
assessing quality of care. First, the validity of structure and 
process indicators remains in question, because the extent to 
which these indicators are linked to outcome is unknown. 
The Institute of Medicine's (lOM; 1991) definition of 
quality care referred to "the degree to which services for 
individuals and populations increase the likelihood of 
desired health outcomes and are consistent with current 
professional knowledge" (p.l). Only those quality indicators 
for which links among structure, process, and outcome exist 
can be referred to as valid (AHCPR, 1995; Wells & Brook, 
1989). 

Unfortunately, structure and process indicators are mostly 
based on expert judgment, which may reflect what we call 
aesthetic quality and may not be associated with improved 
outcomes. Despite their intuitive attractiveness, these 
indicators may have no functional use. For example, the 
setting of a program, such as a spacious and pleasant 
physical environment, might not be tied to a clinical 
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physical environment, might not be tied to a clinical 
outcome as one might theorize. Credentials and provider 
experience might be another example of aesthetic quality. 
Neither of these have been clearly linked to better clinical 
outcomes (Berman & Norton, 1985; Durlak, 1979; Stein & 
Lambert, 1984; Weisz, Weiss, Alicke, & Klotz, 1987). 

Some promising links between service processes and 
clinical outcome have been found for outpatient 
psychotherapy. However, most of this work resulted from 
laboratory research. Little research has been conducted in 
community-based settings. It is our concern that the lack of 
valid indicators may result in meaningless quality assurance 
and continuous quality improvement efforts. 

A second problem with the current assessment of quality 
concerns peer reviews. Peer review approaches to assessing 
quality have been criticized for being unreliable. For 
example, studies have found low inter-rater reliability for 
peer reviews of the quality of outpatient psychotherapy 
(Cohen & Nelson, 1982; Dali & Clairbom, 1982). Based on 
his review of the research, Sechrest (1987) concluded that 
peer review is of limited use for assessing quality. This is 
due to the fact that peer review is based on intuition and 
theoretical biases that may result in erroneous conclusions. 

The unreliability of panel reviews is evident in studies that 
examine factors impacting the evaluation of 
appropriateness. Research has shown, for example, that 
panel composition (Fraser, Pilpel, Kosecoff, & Brook, 

1994) and knowledge of clinical outcome before judgments 
of appropriateness (Caplan, Posner, & Cheney, 1991) have 
been found to influence peer review ratings of 
appropriateness. 

A third problem with current quality assessment approaches 
is that outcome monitoring alone does not provide any 
information about the quality of care that supposedly led to 
those outcomes. Interpreting outcomes should be done with 
great caution, because outcomes may have occurred as a 
result of many influences other than the services provided. 
Threats to internal validity may include history and 
maturation, involvement in nonprofessional programs, or 
natural recovery processes. Whereas Donabedian (1980) 
recognized the need to examine outcomes that can be 
attributed to antecedent health care, mental health services 
researchers often do not attempt to distinguish between 
services-related outcomes and non-services-related changes. 

Quality as Relationship Between Structure, Process, and I 
Outcome 

In order to refocus our attention on the meaningful 
assessment of quality, effort needs to be made to study the | 
validity of quality indicators. In order to further this j 

research, we propose a model whereby quality is viewed j 
strictly as the relationship between quality indicators (i.e., | 

structure and process variables) and quality validators (i.e., 
outcomes: see Figure 11 . This is in contrast to viewing 
quality as existing separately within each component. Given i 
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quality as existing separately within each component. Given 
this conceptualization, the quality of a specific structure or 
process variable is determined by the extent to which it is 
related to some outcome. Given this understanding of 
quality, quality of care research would examine the links 
between quality indicators and validators. For example, 
coordination of services, a structural variable, may be found 
to be significantly associated with decreased symptoms and 
increased functioning. Coordination of services would then 
be viewed as a valid quality indicator of decreased 
symptoms and increased functioning. Interpersonal process 
between the clinician and child/family may be found to be 
significantly associated with satisfaction, but not the other 
outcome domains. Therefore, interpersonal process would 
be viewed as a quality indicator of satisfaction. 
Completeness of paperwork, another process indicator, may 
be found to be unrelated to any indicator of outcome and 
would therefore not be considered a valid quality indicator 
of clinical outcome. 

Advantages of Adopting this Conceptualization of 
Quality 

Adoption of this conceptual framework of quality will 
benefit the field in a number of ways. First, and most 
important, measuring valid quality indicators and making 
changes in response to this data (e.g., through continuous 
quality improvement efforts) will mean better services for 
children and families. Second, rather than making broad 
statements about the quality of a particular service or 
program, providers and evaluators will be able to explicitly 
state what outcomes are associated with specific structural 
or process features. For example, a particular service or 
program will be able to say that they have short waiting 
periods that have been shown to be associated with high 
satisfaction, offer services according to practice guidelines 
that have been found to be associated with better clinical 
outcomes, and follow established procedures to protect 
client rights to confidentiality. Third, distinguishing 
between outcomes associated with the various indicators 
will assist policymakers and administrators in making more 
informed decisions about the utility of certain program 
features over others. For example, improvement efforts 
aimed at altering process indicators associated with 
decreased symptoms and increased functioning might be 
encouraged over those indicators associated with higher 
satisfaction. 

In conclusion, current quality assurance and continuous 
quality improvement efforts are limited in their impact to 
the extent that the quality indicators used are valid. The 
validity of quality indicators is crucial if we are to 
meaningfully examine quality of care. 
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